Question for Step 2 - can automated causal mapping successfully code causal information?: Automated coding was able to identify causal claims made by respondents. The coding was noisy, with 35% dropping at least one quality point, but with no evidence of systematic errors. This level of precision is adequate for sketching out “causal landscapes” but would not be for high-stakes evaluations without additional manual correction. The accuracy can also be substantially improved by getting the AI to revise its work, (see redacted). This procedure still involves the researchers making significant high-level decisions in the formulation of the coding instructions as well as, before analysis, in clustering similar factor labels into groups. We believe this coding approach using genAI represents a significant improvement over the more hard-coded approaches for identifying causal relationships expressed in text (Dunietz 2018)Yang et al., 2022), and provides a more detailed, section-by-section coding which relies less on using AI as a black box to identify themes for initial coding (Jalali & Akhavan 2024) or to identify a global map (Graham 2023).
References
Dunietz (2018). Annotating and Automatically Tagging Constructions of Causal Language.
Graham (2023). Using ChatGPT for Foresight: Futures Wheel. https://medium.com/@christian.graham_49279/using-chatgpt-for-foresight-futures-wheel-8e79eecfe86b.
Jalali, & Akhavan (2024). Integrating AI Language Models in Qualitative Research: Replicating Interview Data Analysis with ChatGPT. https://doi.org/10.1002/sdr.1772.